docs(promotion): add lesson promotion filter and promote the current batch - #21
Conversation
…batch Adds a documented, repeatable filter for deciding whether an instance-level lesson is general (ships to every instance) or specific (stays local), and promotes the batch that passes it. Promoted lessons go to hermes-skill/references/ rather than docs/, because only hermes-skill/, matilde_plugin/ and docker/SOUL* are installed into a deployed agent's data directory. A rule an agent must act on that lives in docs/ never reaches it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…wn fixture The positive control for the instance-particular shape scan carried a real guild ID from the instance this package was promoted from — in a public repo, under a docstring asserting the strings are synthetic. The scan only walked tracked *.md, so every test was green while the leak sat inside the scanner itself. - Replaced with a fabricated 1000000000000000001. - Extended the scan to tracked tests/*.py, so it now reads the file it is written in. - Added an explicit `leak-scan: synthetic` line marker for the fixtures that must contain the forbidden shapes, so the exemption is narrow, greppable and reviewable rather than a whole-file skip. Proved red-then-green both ways: replanting the real ID fails the scan; a real ID on any non-marked line fails it too. 291 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An audit of this PR against its own filter found instance material that should not have travelled into a public package. - deployment-reach.md: drop the C1-C5 next-steps table. It is one instance's operational backlog and fails T2 outright -- no stranger can act on "commit the deployed artifact directory to the instance's private repo". Only C2 survives, rewritten as a design recommendation about the mechanism (a re-apply should be non-destructive or refuse) rather than a task assigned to someone, with the overlay-path point kept as its corollary. - deployment-reach.md: de-urgency the incident passage. "right now", "about thirty" documents and "six weeks" of editing are this instance's timeline, not a general claim. Restated as the mechanism: re-applying against a hand-edited runtime deletes any file the runtime holds that the template does not. - agent-failure-modes.md: replace a real incident date in the rendering example with an obviously generic one. The illustration keeps its shape without carrying our timeline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…omoted The Kept local table claimed the label-taxonomy family/label counts were withheld under T1. They were not: "41 families to 14 classes" ships in hermes-skill/references/evaluation-validity.md as E5's evidence. The PR was rejecting something it had promoted. The rejection list is what calibrates the next contributor, so an inconsistency there is worse than a missing entry -- it teaches the wrong boundary. Reconciled honestly rather than by deleting the E5 evidence: what stays local is the taxonomy's identity and the specific class filter; the raw counts are derived quantities and travel with the rule under T3. Fixed in both places the claim appears (the T1 worked example and the Kept local row). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…enforce The strongest candidate in this batch was not promoted. The incident sits in docs/trustworthy-comparison.md, and docs/ is contributor-facing and not shipped to agents -- a split this very PR adds a test for. So the rule was at rung 1 of the package's own enforcement ladder: available, greppable, citable, and reaching no running agent. That is exactly the recurrence mechanism deployment-reach.md was written about, and it was happening inside the promotion pass that documented it. E6, written generally, in the shipped reference: - A saved model encodes unstated assumptions about input preprocessing -- sampling/resolution, transform, scaling, normalisation scope. Loading it checks none of them and nothing errors when they are violated; the model answers a different question with the same confidence. - Parameters can look identical while preprocessing differs, because the quantity that must match is often derived (hop / sample rate = seconds per frame). Report derived quantities, not the constants. - Write the preprocessing signature into the checkpoint at save time and assert it on load; refuse to run on mismatch. Cheap, and it converts a silent large error into a loud stop. - Never retype a preprocessing function into a new script -- import the one training used. A reimplementation is a new program wearing an old name. - A borrowed checkpoint is the high-risk case: weights you did not train in this script carry no guarantee about this script's preprocessing. - An ablation whose arms differ in more than one respect does not attribute its delta. Say so instead of reporting it. Evidence is bare derived metrics only (publishable under the doc's own T3); no study, species, collaborator, script or split named. Pointer added to SKILL.md beside the existing evaluation block so it is loaded, not merely stored. Also generalises E4's representation example, which read as domain-shaped: a linear-amplitude representation where the field standard is logarithmic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audit against the governing rule, and five editsRe-read this PR against the rule it exists to serve: this repo is the public base package every instance installs. General research-methodology rules belong here; our instance's particulars and to-do list do not. Five things needed fixing, one of them sharp. 1. Cut the instance to-do list —
|
1 participant
What
docs/lesson-promotion-filter.md— four ordered tests (substitution, counterfactual-audience, independence, recurrence-or-mechanism) that decide whether an instance's hard-won lesson is general or instance-local, with worked verdicts on real material including the rejections.hermes-skill/references/:evaluation-validity.md,enforcement-ladder.md,agent-failure-modes.md, each reachable fromSKILL.mdat the point in the workflow where it would be violated.docs/deployment-reach.md— merging is not shipping; traces the four manual steps betweenmainand a running agent and assessesPOST /api/harnesses/:id/usecase/reapplyas the delivery mechanism.tests/test_docs_integrity.py(10 tests) enforcing the mechanical half: internal links resolve, the shipped skill never links outside itself, every shipped reference is pointed at fromSKILL.md, and no tracked markdown carries instance-runtime shapes. Fixes a pre-existing dead link inCONTRIBUTING.md(docs/onboarding-contributors.md→docs/onboarding.md), which is what the link test caught first.Why
Lessons were accumulating in one instance's runtime and in org memory, and only reached the package when someone did it by hand.
promotion-and-upstream.mdalready says how to promote but not what, and "strip the particulars, keep the method" is easy to agree with and hard to apply — the load-bearing cases are rules stated in one domain's vocabulary whose mechanism is domain-free. The filter makes that call explicit and repeatable, so the next lesson has a path up rather than needing this exercise repeated.What was promoted, and what was not
Promoted (each rewritten domain-free; evidence reduced to derived metrics and code shapes):
an optional guard is documentation, and the available/default/gated ladder · a guard should cost a report, not a run · running a guard's tests ≠ proving anything invoked it · prove a negative check can return a positive before trusting it · your metric's ceiling is the reliability of your ground truth · report the tolerance curve, not the point · leakage has a unit (file-disjoint ≠ subject-disjoint) · if a config knob moves the metric more than the claimed effect, you are measuring configuration · the trajectory check (per-artifact validation is structurally blind to reversals) · record your data-selection threshold and show a second cut · pressure trades verification for resolution · default affirmation makes agreement uninformative · a derived value restated on a schedule becomes an observation · a mentioned date is not a deadline without
set_by/type/deliverable· a stated limitation is a claim with a date on it.Deliberately left instance-local: label-taxonomy counts and the class filter, identifier-from-filename parsing, per-tool script paths and flags, corpus/site/collaborator names, chat-platform guild/channel/bot IDs used by a scheduled job, the specific date mistaken for a deadline, and an archival corpus that identifies one unpublished study's design. Each is recorded in the filter doc with the test it failed, because the rejected list is how the next contributor calibrates.
Already promoted, so not re-promoted: "score every arm of a comparison with the same matcher." It is in
docs/trustworthy-comparison.md(#18) — and the instance made that exact error again afterwards. The filter was never the bottleneck; delivery was. That finding is what produceddeployment-reach.md.Verification
python3 -m pytest tests/— 291 passed, 7 skipped (baseline onorigin/mainwas 281 passed, 7 skipped; +10 from the new file). No existing test modified.sanitization: clean (both layers ran)in CI. The semantic layer — the one that detects study particulars, and the one that matters here because this PR moves text from a private context into a public repo — ran under--require-semantic(fails closed) and returnedokfor all 9 content-bearing files. Locally I could only run the deterministic layer (no key on this machine), which was also clean; the CI run is the one to trust.Risk
One finding here is operational and time-sensitive, and is the reason
deployment-reach.mdexists.Re-apply is destructive at directory granularity: with
overwriteset — which re-apply always sets — the install removes the destination artifact directory recursively (rm(destDir, {recursive: true, force: true})) before copying. Calling it against the instance that motivated this PR today would delete ~30 reference documents that exist in no repository anywhere, including the entire batch this PR harvests, and replace the directory with the singleSKILL.mdthe package currently ships. The SOUL is deliberately exempted from re-seeding for exactly this reason; the carve-out has not been extended to skill directories.So: capture the deployed artifact directory into the instance's private repo before anyone calls re-apply.
deployment-reach.mdlists that as C1 with the rest of the sequence.Second, nothing in this PR reaches a running agent on merge. The template registry pins
v0.4.1, which is six commits behindmain; a tag must be cut and the pin bumped. Merging this changes the repository and nothing else — stated here rather than left implied.This PR itself is documentation plus one new test file. No plugin, engine, or CI behaviour changes.